System and method for depth from defocus imaging

ABSTRACT

An imaging system includes a positionable device configured to axially shift an image plane, wherein the image plane is generated from photons emanating from an object and passing through a lens, a detector plane positioned to receive the photons of the object that pass through the lens, and a computer programmed to characterize the lens as a mathematical function, acquire two or more elemental images of the object with the image plane of each elemental image at different axial positions with respect to the detector plane, determine a focused distance of the object from the lens, based on the characterization of the lens and based on the two or more elemental images acquired, and generate a depth map of the object based on the determined distance.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation of, and claims priority to,U.S. non-provisional application Ser. No. 13/272,424, filed Oct. 13,2011, the disclosure of which is incorporated herein by reference.

GOVERNMENT LICENSE RIGHTS

This invention was made with Government support under grant numberHSHQDC-10-C-00083 awarded by the Department of Homeland Security. TheGovernment has certain rights in the invention.

BACKGROUND OF THE INVENTION

Embodiments of the invention relate generally to a system and method fordepth from defocus imaging, and more particularly to a contactlessmulti-fingerprint collection device.

It is well known that the patterns and geometry of fingerprints aredifferent for each individual and are unchanged over time. Thusfingerprints serve as extremely accurate identifiers of an individualsince they rely on un-modifiable physical attributes. The classificationof fingerprints is usually based on certain characteristics such asarch, loop or whorl, with the most distinctive characteristics being theminutiae, the forks, or endings found in the ridges and the overallshape of the ridge flow.

Traditionally, fingerprints have been obtained by means of ink andpaper, where a subject covers a surface of their finger with ink andpresses/rolls their finger onto paper or a similar surface to produce arolled fingerprint. More recently, various electronic fingerprintscanning systems have been developed that obtain images of fingerprintsutilizing an optical fingerprint image capture technique. Suchelectronic fingerprint scanning systems have typically been in the formof contact based fingerprint readers that require a subject's finger tobe put in contact with a screen and then physically rolled across thescreen to provide an optically acquired full rolled-image fingerprint.However, contact-based fingerprint readers have significant drawbacksassociated therewith. For example, in a field environment, dirt, greaseor other debris may build up on the window of contact based fingerprintreaders, so as to generate poor quality fingerprint images.Additionally, such contact-based fingerprint readers provide a means ofspreading disease or other contamination from one person to another.

In recent electronic fingerprint scanning systems, contactlessfingerprint readers capture fingerprints without the need for physicalcontact between a subject's finger and a screen. The goal is to generatea rolled equivalent fingerprint image using a contactless imaging systemin which images are formed by a lens. Conventional imaging provides 2Drepresentation of the object, whereas to generate the rolled equivalentfingerprint, one requires the 3D profile of the finger. For an objectsuch as a finger, some parts of the object are in focus and some aredefocused when imaged with a shallow depth of field imaging system.Typically, an in-focus region is a region of an object that is in assharp as possible focus, and conversely defocus refers to a lack offocus, the degree of which can be calculated between two images. Knownsystems may generate a depth map of the object using either a depth fromfocus (DFF) or a depth from defocus (DFD) algorithm.

In one system, a contactless fingerprint scanning system acquires animage of the finger by utilizing a structured light source, and a 3Dimage is generated using a DFF algorithm. In a DFF algorithm, as anexample, many measurements are made at various focal plane positions andthe many measurements are used to generate a depth map. Typically, thevarious focal plane positions are obtained by either physical movementof the object or lens, or by adjustment of the focal plane (using knowntechniques or using one or more birefringent lenses producing focalshifts at different polarization angles passing therethrough). DFF-basedsystems, however, typically require many measurements to be obtained andalso may include adjustment of the focal plane to focus on the object,as well as a structured light source.

For a given object, the amount of defocus depends on at least twoparameters: 1) a distance of the object to the lens, and 2) the lenscharacteristics. If the second parameter (i.e., the lenscharacteristics) is known, and the system can accurately measure anamount of defocus, then the object distance can be determined. Suchforms the basis of known DFD algorithms.

Thus, in some contactless finger print readers, the system acquires animage of the finger by utilizing a white light source, and a 3D image isgenerated using a DFD algorithm. In a DFD algorithm, a defocus functionacts as a convoluting kernel with the fingerprint, and the most directway to recover it is through the frequency domain analysis of obtainedimage patches. Essentially, as the amount of defocus increases, theconvolving kernel's width decreases, resulting in elimination of highfrequency content.

DFD algorithms typically start with an assumption of a simplifiedGaussian or pillbox estimator for a point spread function (PSF),building up on a polychromatic illumination assumption. Typically, anobject point, when imaged, will look like a bell curve rather than asharp point. The function describing the shape of the bell curves iscalled the ‘PSF’, and the shape of the PSF on an image detector dependson the distance of the object point to the lens, as well as internallens characteristics. Thus, these assumptions simplify the mathematicalderivations and provide a convenient approach to DFD. The extent towhich such assumptions hold depends on the particular imaging system andillumination condition. For highly corrected imaging optics and whitelight illumination, the PSF resembles a Gaussian or a pillbox andassuming so typically generates a depth estimator with a reasonableerror. However, it can be shown that depth estimation based on DFD ishighly sensitive to proper determination of PSF structure, and applyingDFD based on Gaussian (or pillbox) PSF models to an imaging system wherePSF departs from this assumption results in unreliable depth estimates.That is, the simplified model does not adequately describe physical lensbehavior when there is a high degree of aberration, when a lens has asmall depth-of-field compared to object size, when quasi-monochromaticlight is used (such as an LED), or when monochromatic light is used(such as a laser), as examples. Thus, known DFD systems fail to estimateobject distance and fail to accurately reproduce a fingerprint in acontactless system.

Therefore, it would be desirable to design a system and method ofacquiring fingerprints in a contactless application that accounts forlens imperfections.

BRIEF DESCRIPTION OF THE INVENTION

Embodiments of the invention are directed to a system and method forcontactless multi-fingerprint collection.

According to one aspect of the invention, an imaging system includes animaging system includes a positionable device configured to axiallyshift an image plane, wherein the image plane is generated from photonsemanating from an object and passing through a lens, a detector planepositioned to receive the photons of the object that pass through thelens, and a computer programmed to characterize the lens as amathematical function, acquire two or more elemental images of theobject with the image plane of each elemental image at different axialpositions with respect to the detector plane, determine a focuseddistance of the object from the lens, based on the characterization ofthe lens and based on the two or more elemental images acquired, andgenerate a depth map of the object based on the determined distance.

According to another aspect of the invention, a method of imagingincludes mathematically characterizing a lens as a mathematicalfunction, acquiring two or more elemental images of an object with animage plane of the object at differing axial positions with respect to adetector, determining a first focused distance of the image plane to theobject such that the image plane is located at the detector, based onthe mathematical characterization of the lens and based on the first andsecond elemental images, and generating a depth map of the object basedon the determination.

According to yet another aspect of the invention, a non-transitorycomputer readable storage medium having stored thereon a computerprogram comprising instructions which, when executed by a computer,cause the computer to derive a pupil function of a lens, acquireelemental images of an object at different locations of an image planeof the object with respect to a detector, determine where to place theimage plane of the first patch of the object based on the pupil functionand based on the acquired elemental images of the first patch of theobject, and generate a depth map of the object based on thedetermination.

Various other features and advantages will be made apparent from thefollowing detailed description and the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings illustrate preferred embodiments presently contemplated forcarrying out the invention.

In the drawings:

FIG. 1 illustrates a typical fingerprint spectrum.

FIG. 2 illustrates an exemplary radial frequency spectrum of a typicalfingerprint image.

FIG. 3 illustrates a first radial spectrum and a second radial spectrumfor images having different levels of blur.

FIG. 4 illustrates an effect of blurring one image using an exemplaryGaussian kernel.

FIG. 5 illustrates coordinate systems used to identify planes in thelens in reference to embodiments of the invention.

FIG. 6 illustrates a method of correcting an image usingdepth-from-defocusing (DFD), according to the invention.

DETAILED DESCRIPTION

According to the invention, a mathematical model is used that governslens behavior. The model is affected by object distance and physicalcharacteristics of the lens (i.e., aberrations, focal length, etc. . . .). Information from focus planes (DFF) and from an amount of defocus(DFF) is combined to yield a depth map. Following is a description of analgorithm for a contactless fingerprint imaging system according toembodiments of the invention. However, the invention is not limited tosuch a system and it is contemplated that the disclosed invention may beapplicable to any imaging system that uses passive depth estimation froma set of slightly defocused images such as 3D microscopic profilometryfor inspection in industrial applications, 3D borescope imaging, 3Din-situ medical imaging, 3D consumer cameras (with proper focus shiftinglenses), passive imaging for 3D target recognition (defense or securityindustries), and the like.

FIG. 1 illustrates a typical fingerprint spectrum 100 that may beobtained from a common fingerprint and generated using a Fouriertransform, as known in the art. In a typical fingerprint and in thefrequency domain it is evident that the patterns exhibit a distinctperiodicity that is represented in the spectral data as an abruptconcentration or halo 102. Hence, useful information, in terms of depthestimation, can be extracted in fingerprint imaging based on this knownperiodicity.

The DC component 104 (near the center of the spectral data of FIG. 1)may be separated from the higher frequency halo 102 that is symmetricand can be attributed to the fingerprint marks. The spectrum can betransformed to polar coordinates to generate a projection on a radialfrequency axis using the following:

$\begin{matrix}{{{I_{p}( f_{r} )} = {\int_{- \pi}^{+ \pi}{{I_{p}( {f_{r},\theta} )}\ {\mathbb{d}\theta}}}},} & {{{Eqn}.\mspace{14mu} 1};}\end{matrix}$where f_(r) denotes the radial frequency and I_(p) (f_(r), θ) denotesthe spectrum in polar coordinates.

FIG. 2 illustrates an exemplary radial frequency spectrum 150 of atypical fingerprint image. The actual fingerprint marks exhibitthemselves through a hump 152 in spectrum 150. This is in contrast toexpected behavior of natural images (i.e., those not having a generallysymmetric pattern such as in a fingerprint), which may be modeled by anexponential decay of the form I_(p)(f_(r))=1/f_(r) ^(α). Typically, themost visible detailed features of a fingerprint image are the ridges andgrooves, and it is the defocus of these features that is measured,according to embodiments of the invention.

Conventional DFD methods assume a certain form for a point spreadfunction (PSF) of the lens, resulting in a use of known functions suchas a Gaussian or a Pillbox function in lieu of PSF. However, when thereal PSF shape departs significantly from assumptions, conventional DFDalgorithms tend to provide poor results. That is, for an object like afingerprint, having hump 152 in spectrum 150 as illustrated in FIG. 2,using a known and conventional blurring kernel can cause a conventionalDFD method to break down, thus failing to provide a satisfactory finaldepth image using DFD.

For example, in order to illustrate that known DFD methods using aGaussian or Pillbox function are not proper estimates for the blurringprocess, a patch of one image may be blurred with kernels of varioussize and shape, and the resulting image can be compared with a secondimage obtained by the imaging system. Beginning with a plot 200,referring to FIG. 3, a first radial frequency spectrum 202 and a secondradial frequency spectrum 204 having differing levels of blur areillustrated. Thus, according to conventional DFD methods, known blurringkernels could be applied to, for instance, first radial frequencyspectrum 202 in order to reproduce second radial frequency spectrum 204.The objective is to understand if, for instance, a Gaussian blurringkernel can in fact transform the first image, from which first radialfrequency spectrum 202 is derived, to the second image, from whichsecond radial frequency spectrum 204 is derived. Referring to FIG. 4, inone example first radial frequency spectrum 202 is blurred 206 with aGaussian kernel with a 0.9 pixel standard deviation width. As can beseen in FIG. 4, the spectrum of blurred image 206 departs from theactual image 204 captured by the imaging system. Similar behavior can beshown for different standard deviations of the Gaussian kernel, andsimilar behavior can also be shown for other blurring kernels, such as apillbox kernel with different standard deviations.

Thus, it can be observed that neither the Gaussian nor the pillboxblurring kernels are able to acceptably reproduce one defocused imagefrom another image. As such, according to the invention, informationabout the PSF of the lens is experimentally or empirically obtainedinstead of using a theoretical kernel such as a Gaussian or a pillbox.As seen in the exemplary FIGS. 3 and 4, the high frequency contentappears to be present in both images, which can be attributed toelectronic and quantization noise. As a result, it is unlikely that highfrequency content of images can be relied upon for DFD calculations.Thus, a low pass pre-filter can be used to remove the high frequencyportion of the spectrum before further processing.

Accordingly, if an imaging lens does not exhibit a typical Gaussian,pillbox, or other analytical form PSF, the required information can bederived empirically or through pupil map for designing a reliableDFD-based depth estimator, according to the invention. Referring to FIG.5, a framework 300 includes an object plane 302, an exit pupil 304(which corresponds to a location of a lens 306), an image plane 308, anda detector plane 310 (of, for instance, a charge-coupled device, orCCD). Photons emanate from object plane 302, pass through exit pupil304, and form a clean image at image plane 308 which, depending ondistances and characteristics of the imaging system, may not coincidewith the location of the detector plane 310. Thus, system 300 representsan imaging system which can change its focal length.

The imaging lens characteristics are reduced to its exit pupil.Typically, a pupil function map (or pupil map) is a wavefront at theexit pupil of the imaging system for a given object position in space.As known in the art, as distance z_(o) between object plane 302 and exitpupil 304 is varied, image plane 308, at a distance z_(i) from exitpupil 304 likewise varies. As such and for clarification, it is desiredto know the value of z_(o) that will place image plane 308 coincidentwith detector plane 310 such that a clean or sharply focused image of anobject at object plane 302 may be obtained. According to one embodimentand as illustrated, lens 306 may be positioned on a moveable stage 312that may itself be translatable along a translation axis 314, which maybe used to obtain a plurality of elemental images of an object that ispositioned at object plane 302. Typically, an elemental image is asingle image taken with a specific lens setting and configuration (i.e.,focal length). Distance z_(o) may be altered in other fashions accordingto the invention. For instance, the object at object plane 302 mayinstead be translated by an object translator 316 that can translateobject plane 302 along translation axis 314. Further, distance z_(o) mayalso be altered, according to the invention, using other techniquesknown in the art that include but are not limited to a variable pathwindow, a prism, a piezo-electric translator, a birefringent optic, andthe like. As such, distance z_(o) may be actually and physicallyaffected by physical movement of the object and/or the lens, or distancez_(o) may be virtually affected by altering an apparent distancetherebetween by using, for instance, the variable path window, theprism, or the birefringent optic, as examples.

Referring now to FIG. 6, a method of obtaining a depth of an object isillustrated therein. And, as stated, although embodiments of theinvention are described as they relate to acquisition of fingerprintimages, it is contemplated that the invention described herein isapplicable to a broader array of imaging technologies. For instance, inother applications where a DFD technique is not optimized because knownkernels do not adequately represent properties of the imaging system,such as a PSF of the lens.

FIG. 6 illustrates a technique or method 400, according to theinvention, having an offline component 402 and an online component 404.Generally, offline component 402 includes steps for empiricallycharacterizing a lens, such as lens 306 of illustrated in the system ofFIG. 5. Online component 404 includes acquisition of images, andmanipulation thereof by taking into account the characterization of thelens and the PSF or pupil function derived from offline component 402.

The overall technique 400 is described as follows: Referring back toFIG. 5, a pupil function is represented as p(x,y) and PSF with h(x,y)which can be found through lens design software packages or empiricallythrough various methods including interferometry. Note that the pupilfunction and PSF on the imaging plane have the following relationship:h(u,v;γ,z _(o))=ℑ{p(−λz _(i) x,−λz _(i) y;γ,z _(o))};  Eqn. 2,where ℑ{•} denotes Fourier transformation and γ denotes a particularfocal setting on the lens, and λ is the illumination wavelength. As thescaled version of Fourier pairs are related through Fourier transformas:

$\begin{matrix}{{{{p( {x,y} )} \cdot}\overset{\mathfrak{J}}{rightarrow}{P( {f_{x},f_{y}} )}}{{{{p( {{{- \lambda}\; z_{i}x},{{- \lambda}\; z_{i}y}} )} \cdot}\overset{\mathfrak{J}}{rightarrow}{\frac{1}{\lambda\; z_{i}}{P( {\frac{f_{x}}{{- \lambda}\; z_{i}},{\frac{f_{y}}{{- \lambda}\; z_{i}};\gamma},z_{o}} )}}};}} & {{{Eqn}.\mspace{14mu} 3},}\end{matrix}$one can write:

$\begin{matrix}{{{h( {u,{v;\gamma},z_{o}} )} = {\frac{1}{\lambda\; z_{i}}{P( {{f_{x} = {{- \lambda}\; z_{i}u}},{{f_{y} = {{- \lambda}\; z_{i}v}};\gamma},z_{o}} )}}};} & {{Eqn}.\mspace{14mu} 4.}\end{matrix}$However, because the detector plane does not coincide with the imageplane in general, a quadratic phase factor (defocus) can be used tocompensate the pupil function and account for this distance:

$\begin{matrix}\begin{matrix}{{h( {s,{t;\gamma},z_{o}} )} = {{\mathfrak{J}}\{ {{\mathbb{e}}^{j\;{k{({x^{2} + y^{2}})}}}{p( {{{- \lambda}\; z_{i}x},{{{- \lambda}\; z_{i}y};\gamma},z_{o}} )}} \}}} \\{= {{\mathfrak{J}}\{ {p^{\prime}( {{{- \lambda}\; z_{i}x},{{{- \lambda}\; z_{i}y};\gamma},z_{o}} )} \}}} \\{{= {\frac{1}{\lambda\; z_{i}}{P^{\prime}( {{{- \lambda}\; z_{i}s},{{{- \lambda}\; z_{i}t};\gamma},z_{o}} )}}};}\end{matrix} & {{{Eqn}.\mspace{14mu} 5},}\end{matrix}$where

$k = {\frac{\pi}{\lambda}( {\frac{1}{z_{i}} - \frac{1}{z_{d}}} )}$is related to the distance between image plane and detector plane andvanishes when imaging condition holds, i.e. z_(i)=z_(d).

Next, the image formed on the detector can be written as a convolutionbetween the PSF and an ideal image, such as:i _(γ)(s,t)=i ₀(s,t)

h(s,t;γ,z _(o))I _(γ)(f _(s) ,f _(t))=I ₀(f _(s) ,f _(t))×H(f _(s) ,f _(t) ;γ,z_(o));  Eqn. 6.By invoking the duality principle of Fourier transformation, it can beshown that:

$\begin{matrix}{{{{p( {s,t} )} \cdot}\overset{\mathfrak{J}}{rightarrow}{P( {f_{s},f_{t}} )}}{{{P( {s,t} )} \cdot}\overset{\mathfrak{J}}{rightarrow}{2\;\pi\;{p( {{- f_{s}},{- f_{t}}} )}}}{{{{P( {{{- \lambda}\; z_{i}s},{{- \lambda}\; z_{i}t}} )} \cdot}\overset{\mathfrak{J}}{rightarrow}{\frac{2\;\pi}{\lambda\; z_{i}}{p( {\frac{f_{s}}{\lambda\; z_{i}},\frac{f_{t}}{\lambda\; z_{i}}} )}}};}} & {{Eqn}.\mspace{14mu} 7.}\end{matrix}$Thus,

$\begin{matrix}{{{H( {f_{s},{f_{t};\gamma},z_{o}} )} = {\frac{2\pi}{\lambda^{2}z_{i}^{2}}{p^{\prime}( {\frac{f_{s}}{\lambda\; z_{i}},{\frac{f_{t}}{\lambda\; z_{i}};\gamma},z_{o}} )}}};} & {{Eqn}.\mspace{14mu} 8.}\end{matrix}$The image spectra can be re-written as:

$\begin{matrix}{{{I_{\gamma}( {f_{s},f_{t}} )} = {\frac{2\pi}{\lambda^{2}z_{i}^{2}}{I_{0}( {f_{s},f_{t}} )} \times {p^{\prime}( {\frac{f_{s}}{\lambda\; z_{i}},{\frac{f_{t}}{\lambda\; z_{i}};\gamma},z_{o}} )}}};} & {{{Eqn}.\mspace{14mu} 9},}\end{matrix}$and the spectral ratio as:

$\begin{matrix}{{\frac{I_{\gamma_{1}}( {f_{s},f_{t}} )}{I_{\gamma_{2}}( {f_{s},f_{t}} )} = \frac{p^{\prime}( {\frac{f_{s}}{\lambda\; z_{i}},{\frac{f_{t}}{\lambda\; z_{i}};\gamma_{1}},z_{o}} )}{p^{\prime}( {\frac{f_{s}}{\lambda\; z_{i}},{\frac{f_{t}}{\lambda\; z_{i}};\gamma_{2}},z_{o}} )}};} & {{{Eqn}.\mspace{14mu} 10},}\end{matrix}$which holds point for point for different (f_(s),f_(t)) and can beexpressed in Polar coordinates as:

$\begin{matrix}{{\frac{I_{\gamma_{1}}^{p}( {\rho,\theta} )}{I_{\gamma_{2}}^{p}( {\rho,\theta} )} = \frac{p_{p}^{\prime}( {\frac{\rho}{\lambda\; z_{i}},{\theta;\gamma_{1}},z_{o}} )}{p_{p}^{\prime}( {\frac{\rho}{\lambda\; z_{i}},{\theta;\gamma_{2}},z_{o}} )}};} & {{{Eqn}.\mspace{20mu} 11},}\end{matrix}$where p′(f_(s),f_(t))

p′_(p)(ρ,θ) results in p′(af_(s),af_(t))

p′_(p)(aρ,θ). Script p denotes Polar coordinates.The pupil function can be expressed with Zernike polynomials, in oneexample, as:p′ _(p)(ρ,θ;γ,z _(o))=W ₁₁ ^(γ,z) ^(o) ρ cos θ+W ₂₀ ^(γ,z) ^(o) ρ² +W ₄₀^(γ,z) ^(o) ρ⁴ +W ₄₁ ^(γ,z) ^(o) ρ³ cos θ+W ₂₂ ^(γ,z) ^(o) ρ² cos²θ,  Eqn. 12.Zernike polynomials are a set of polynomial functions, as illustrated inEqn. 12, that can be used to describe a wavefront efficiently. They actas basis functions to describe a more complex function. It iscontemplated, however, that the invention is not limited to expressionof the pupil function with Zernike polynomials, but that otherfunctions, such as Abbe formulation may be used.Substituting in Eqn. 12 results in:

$\begin{matrix}{{\frac{I_{\gamma_{1}}^{p}( {\rho,\theta} )}{I_{\gamma_{2}}^{p}( {\rho,\theta} )} = \frac{\begin{matrix}{{W_{11}^{\gamma_{1},z_{o}}\cos\;\theta} + {{W_{20}^{\gamma_{1},z_{o}}( {\lambda\; z_{o}} )}^{2}\rho} + {W_{40}^{\gamma_{1},z_{o}}\rho^{3}} +} \\{{W_{41}^{\gamma_{1},z_{o}}\lambda\; z_{o}\rho^{2}\cos\;\theta} + {{W_{22}^{\gamma_{1},z_{o}}( {\lambda\; z_{o}} )}^{2}{{\rho cos}\;}^{2}\theta}}\end{matrix}}{\begin{matrix}{{W_{11}^{\gamma_{2},z_{o}}\cos\;\theta} + {{W_{20}^{\gamma_{2},z_{o}}( {\lambda\; z_{o}} )}^{2}\rho} + {W_{40}^{\gamma_{2},z_{o}}\rho^{3}} +} \\{{W_{41}^{\gamma_{2},z_{o}}\lambda\; z_{o}\rho^{2}\cos\;\theta} + {{W_{22}^{\gamma_{2},z_{o}}( {\lambda\; z_{o}} )}^{2}{{\rho cos}\;}^{2}\theta}}\end{matrix}}};} & {{{Eqn}.\mspace{14mu} 13},}\end{matrix}$which is a polynomial with focal setting dependent coefficients and canbe written in shorthand as:

$\begin{matrix}{{{\hat{z}}_{o} = {\underset{z_{o} \in {\lbrack{z_{\min},z_{\max}}\rbrack}}{\arg\;\min}{{\frac{I_{\gamma_{1}}^{p}( {\rho,\theta} )}{I_{\gamma_{2}}^{p}( {\rho,\theta} )} - \frac{p_{p}^{\prime}( {{{\rho/\lambda}\; z_{i}},{\theta;\gamma_{1}},z_{o}} )}{p_{p}^{\prime}( {{{\rho/\lambda}\; z_{i}},{\theta;\gamma_{2}},z_{o}} )}}}}};} & {{Eqn}.\mspace{14mu} 14.}\end{matrix}$

Referring to Eqn. 13, offline calculation 402 provides the values of thesecond fraction, and the elemental images acquired via online component404 can be processed (Fourier Transformed) to calculate the firstfraction, according to the invention. The minimization strategy,according to the invention, is then to find object distance z_(o) suchthat the difference between the two fractions vanishes. This process isdone for many points on the finger to map out the surface.

As stated, offline component 402 according to the invention includescharacterization of the lens using a series of mathematical steps asdiscussed hereinbelow. In a spectral domain DFD algorithm, the Fouriertransform of the intensity distribution on the CCD for a given pointsource needs to be known. As shown in FIG. 5, the image plane and theCCD plane do not coincide and hence the simple Fourier relationshipbetween the PSF and pupil function is not valid. However, angularspectrum propagation between the pupil function and the CCD plane can beused to calculate the Fourier transform of light distribution on the CCDplane (angular spectrum at (x,y) plane) based on the Fourier transform(angular spectrum) of the pupil function (adjusted with an additionalquadratic phase). The following equations show the process:Need: ℑ{I(x,y)}∝AS(x,y)AS(x,y)∝AS(ξ,η)×e ^(jφ(ξ,η))φ(ξ,η)=f(ξ,η,z _(d))AS(ξ,η)=AS _(sph)(ξ,η)

AS _(ab)(ξ,η);  Eqns. 15.AS_(sph)(ξ,η): can be found analytically (avoid aliasing)very high frequency at the perphery of exit pupilAS_(ab)(ξ,η)=g(W_(ab)): can be computed based on ZernikesW_(ab): aberration (varies by object depth)

Referring to FIG. 5, the schematic shows the coordinate systems at exitpupil, CCD and image planes as well as typical sizes for a fingerprintlens. The distance z_(d) between the exit pupil and CCD plane is fixed,however the distance between the exit pupil and the image plane changesdepending on the object location and lens focal configuration. The sizeof exit pupil of lens 306 varies slightly for different objectdistances.

In order to calculate the Fourier transform of the pupil function, avery large (for example, 35000×35000) discrete Fourier transform (DFT)calculation is needed, which can be prohibitive. This is due to the factthe reference spherical wavefront exhibits rapid phase fluctuations atthe edge of the pupil. To calculate the angular spectrum of such afield, the spatial sampling should satisfy Nyquist criteria. Thefollowing calculations show what spatial sampling period (and size ofmatrix) is, according to one example:

The maximum cosine angle of the planar wavefront at the edge of pupil(D=32 mm) representing the reference sphere focusing at z_(f)=55 mm(pupil to image point distance) is:

$\begin{matrix}{{\alpha_{\max} = {{\cos(\omega)} = {\frac{D/2}{\sqrt{( {D/2} )^{2} + z_{f}}} = {\frac{16}{57.2} = 0.28}}}};} & {{Eqn}.\mspace{14mu} 16.}\end{matrix}$which according to relationship α=λf_(ξ) suggests:max(f _(ξ))=α_(max)/λ=0.28/(0.52×10⁻³)=538 1/mm;  Eqn. 17.According to Nyquist rate, capturing this frequency requires a spatialsampling interval of

$d_{\xi} = {\frac{1}{2\;{\max( f_{\xi} )}} = {0.93\mspace{14mu}\mu\; m}}$or about 35,000 samples of wavefront across the 32 mm diameter. As such,the DFT should then operate on a 35,000×35,000 matrix, which may beimpractical, and which may result in undersampling. Thus, the angularspectrum at pupil function may be calculated indirectly.

The aberration part of the wavefront is typically not high frequency andits angular spectrum can be calculated through DFT. This suggestsbreaking down the calculation of the total pupil wavefront angularspectrum into two problems:

-   -   1. Calculate the angular spectrum of the wavefront aberration        through DFT.    -   2. Directly compute the angular components (planar wavefronts)        of the reference spherical wave at predetermined frequencies.        Since we know exactly what these planar wavefronts are, we can        calculate them at any position on the pupil without introducing        aliasing caused by DFT.        The sampling across the pupil can be relatively sparse (for        example, 128×128). In this example, lens aberrations are not        high frequency, thus can be captured with n_(ab) samples in both        directions. For n_(ab)=256, or d_(ξ)=D/n_(ab)=0.125 mm, this        leads to maximum frequency of max(f_(ξ))=½d_(ξ)=4 mm⁻¹.

As known in the art, the angular components can be directly calculatedfor each directional cosine pair (α, β). The plane wave component onpupil plane at position (ξ,η) can be written as:

$\begin{matrix}{{{\exp( {{- j}\;{\overset{harpoonup}{k} \cdot \overset{harpoonup}{r}}} )} = {\exp( {{- j}\frac{2\pi}{\lambda}( {{\alpha\xi} + {\beta\eta}} )} )}};} & {{{Eqn}.\mspace{14mu} 18},}\end{matrix}$where there is a map that converts any (α, β) pair to pupil coordinates(ξ,η). This relationship is defined as:

$\begin{matrix}{{{\xi = {z_{f}\frac{1 - \sqrt{1 - {2\alpha^{2}}}}{\alpha}}};}{and}} & {{{Eqn}.\mspace{14mu} 19},} \\{{\eta = {z_{f}\frac{1 - \sqrt{1 - {2\beta^{2}}}}{\beta}}};} & {{Eqn}.\mspace{14mu} 20.}\end{matrix}$The equations that map frequency to directional cosines include:α=λf _(ξ);β=λf _(η).  Eqn. 21, and

Thus, for any given discrete grid of (f_(ξ),f_(η)), the plane wavecomponent can be calculated through equations above. This approach canbe taken to directly calculate the angular spectrum at a predefinedfrequency grid that extends to the maximum frequency present on thereference sphere. Because maximum frequency in the present example ismax(f_(ξ))=538 mm⁻¹, a frequency grid with 2000 elements is included ineach direction that covers a [−538,+538] mm⁻¹ region. Angular componentscalculated on this grid will thus be free from aliasing.

The next step is to do the convolution between the aberration wavefrontand spherical wavefront angular frequencies. Once both referencewavefront and aberration angular spectra are calculated, they can beconvolved to arrive at the total wavefront angular spectrum:AS(ξ,η)=AS _(sph)(ξ,η)

AS _(ab)(ξ,η);  Eqn. 22.

Thus, according to the invention and referring back to FIG. 6, offlinecomponent 402 includes, at a high-level, the step of characterizing thelens 406 and mapping the lens as a mathematical function 408. Offlinecomponent 402 may be characterized as a calibration step, performedonce, that characterizes a lens thoroughly and is done through a pupilmap function which describes the amount of aberration for every point inobject space. The pupil function changes for objects at differentlocations. The results of offline component 402 thus provide acharacterization of a lens which, as stated, result in the coefficientsillustrated in the second fraction of Eqn. 13. More generally,mathematical function 408 may be obtained as a general equation thatincludes the use of pupil functions, as illustrated in Eqn. 11. However,according to one embodiment, the pupil function is mathematicallydescribed as a pupil function map through Zernicke coefficients as inEqn. 13. As such, the lens is characterized based on its response topoint sources in different locations in a volume of interest, andcharacterization tables may be generated in a table that maps thedistance of the object to a set of parameters which can be measured fromimages during the online process, and based on the mathematicaldescription disclosed herein.

Online component 404, includes a series of high-level steps consistentwith the mathematical description above. Online component 404 begins byacquiring two or more elemental images 410 of an object for which it isdesired to generate a depth map. A patch of the object is selected atstep 412, and best focus planes are estimated at step 412 using, forinstance, a known DFF method or algorithm, out of the elemental images.Once the best focus planes are estimated, a power spectral ratio betweenelemental images is obtained at step 416, which will thereby be used tocompare to a ratio of the lens function that was obtained correspondingto the same elemental image locations, consistent with Eqn. 11. At step418, object distance is assumed and at step 420 a function ratio iscalculated, based on the lens function obtained at step 408 and based onthe assumed object distance from step 418. At 420, as well, the ratiosare compared, consistent with Eqn. 11, and at step 422 it is determinedwhether the ratios are within a threshold. If not 424, then iterationcontinues and object distance assumptions are revised at step 426, andcontrol returns to step 420 to be compared, again, to the power spectralratio obtained at step 416.

Thus, according to the invention, elemental images are obtained, bestfocus planes are estimated using a known technique (DFF), and a powerspectrum ratio is calculated. The mapped function is calculated thatcorresponds to each of the elemental functions, but based on anassumption of an object distance as a starting point. A ratio of themapped function is calculated that corresponds to the elemental images,as well as a ratio of the elemental images themselves. Iteration therebyincludes revision of the mapped function ratio by revising the assumedobject distance, which continues until the two ratios compare to areasonable threshold. In summary, a ratio of pupil functions at twodifferent lens settings (e.g., focal lengths) is equal to the ratio ofthe power spectrum between the two images formed by the two lenssettings. The distance z_(o) at which the ratio of the power spectrumbetween two best focus elemental images (which can be found by DFF,independent of z_(o)) is closest to the ratio of pupil functions at anobject distance equal to z_(o). This distance z_(o) is the estimateddistance of the object from the lens.

Referring still to FIG. 6, once the ratios are acceptably close 428,then a final distance is obtained at step 430 for the patch selected atstep 412. At step 432 a determination is made as to whether additionalpatches will be assessed. If so 434, then control moves back to step412, another patch is selected, and the process repeats for the newlyselected patch. However, if no additional patches 436, then the processends at step 438 where a complete depth map is generated.

According to additional embodiments of the invention, the contactlessmulti-fingerprint collection device is configured to acquire fingerprintdata for the fingers of the subject without the subject's hand being ina stationary position, but rather being moved (i.e., swiped or waved)through an imaging volume. That is, rather than guiding the subject toplace their fingers in a specified image capture location, thecontactless multi-fingerprint collection device acts to track a locationof the subject's fingers and cause the image capture device(s) toacquire images of the fingers.

According to embodiments of the invention, one or more positioningverification devices may include devices (e.g., overhead camera) thatfunction as tracking devices that are used to verify and track movementof a subject's hand within an imaging volume for purposes of controllingthe image capture devices. That is, a field-of-view and focus depth ofeach image capture device can be independently set based on a movementand placement of the subject's hand/fingers as tracked by trackingdevices, so as to enable following of individual fingertips. The movingof the field-of-view of each image capture device can be accomplishedvia a mechanical actuation of one or more elements or via anelectronic/digital controlling of each image capture device. Forexample, in an embodiment where one or more elements are mechanicallyactuated to move the field-of-view, a mirror positioned adjacent theimage capture device could be rotated or a lens element could be movedin order to shift the field-of-view of the image capture device. In anembodiment where electronic or digital controls are implemented, asensor in the image capture device (i.e., camera sensor) could becontrolled to shift the field-of-view of the image capture device.

Various methods may be used to register the image. As used hereinregistration refers to a process of transforming the different images ofa single subject into one coordinate system. In the context of afingerprint, registered images are derived from the captured images ofthe fingerprint. The registered images have the same scale and featureposition.

In order to ensure the features from the multiple shifted images areapproximately registered, a telecentric lens system is also commonlyused that maintains magnification within a narrow range. However, asknown in the art, the addition of a telecentric aperture inherentlyincreases the f-number and may result in an excessive depth-of-field.

In certain registration embodiments, registration may use a geographicinformation system (GIS) employing ortho-rectification.Ortho-rectification is a process of remapping an image to remove theeffect of surface variations and camera position from a normalperspective image. The resultant multiple images are perspectivecorrected projections on a common plane, representing no magnificationchanges with a pixel to pixel correspondence. In certain embodiments,ortho-rectification may comprise un-distorting each captured image using3D calibration information of the image capture device, and projectionof the image onto one plane.

Once the images are registered, image fusion is used to create a singlehigh-resolution image from the multiple images of the same target.Generally, image fusion is the procedure of combining information frommultiple images into a single image whereas in the said embodiment thisinformation relate to the local, spatial focus information in eachimage. The re-fused image would desirably appear entirely in-focus whilethe source images are in-focus in different, specific regions. This maybe accomplished by using selected metrics. These metrics are chosenbased on the fact that the pixels in the blurred portions of an imageexhibit specific different feature levels, in comparison to those pixelsthat are in good focus. For example, focused images typically containhigher frequencies while blurred images have lower frequency components.

In certain embodiments, certain metrics may be used to compute the levelof focus for each pixel in each separately obtained image of thefingerprint. The separate images are then normalized and combined usinga weighted combination of the pixels to obtain a single fused orcomposite image. Thus, for each of the acquired images, the region ofinterest is determined by image segmentation. From the different metricsthe focus at each location in the image is calculated as a weightedcombination of features, then the images are combined using said localweighted combination of the features.

Upon generation of a composite image of a fingerprint, a contour map or“depth map” of the composite image for each of the plurality offingerprints is calculated/generated using the disclosed depth fromdefocus (DFD) algorithm. The depth from focus analysis/calculation is animage analysis method combining multiple images captured at differentfocus distances to provide a 3D map correlating in-focus locations ineach image with a known focus distance the specific image was capturedat.

In order to match the fingerprint images captured to standard databasesbased upon 2D data capture, the 3D model obtained from the disclosed DFDalgorithm may be used to generate an unrolled 2D image. The model usedsimulates the image distortions corresponding to the reverse of theprojection of the fingerprint surface on a two-dimensional projectionobtained in a contact method.

Therefore, according to one embodiment of the invention, an imagingsystem includes a positionable device configured to axially shift animage plane, wherein the image plane is generated from photons emanatingfrom an object and passing through a lens, a detector plane positionedto receive the photons of the object that pass through the lens, and acomputer programmed to characterize the lens as a mathematical function,acquire two or more elemental images of the object with the image planeof each elemental image at different axial positions with respect to thedetector plane, determine a focused distance of the object from thelens, based on the characterization of the lens and based on the two ormore elemental images acquired, and generate a depth map of the objectbased on the determined distance.

According to another embodiment of the invention, a method of imagingincludes mathematically characterizing a lens as a mathematicalfunction, acquiring two or more elemental images of an object with animage plane of the object at differing axial positions with respect to adetector, determining a first focused distance of the image plane to theobject such that the image plane is located at the detector, based onthe mathematical characterization of the lens and based on the first andsecond elemental images, and generating a depth map of the object basedon the determination.

According to yet another embodiment of the invention, a non-transitorycomputer readable storage medium having stored thereon a computerprogram comprising instructions which, when executed by a computer,cause the computer to derive a pupil function of a lens, acquireelemental images of an object at different locations of an image planeof the object with respect to a detector, determine where to place theimage plane of the first patch of the object based on the pupil functionand based on the acquired elemental images of the first patch of theobject, and generate a depth map of the object based on thedetermination.

A technical contribution for the disclosed method and apparatus is thatit provides for a computer implemented system and method for depth fromdefocus imaging, and more particularly to a contactlessmulti-fingerprint collection device.

One skilled in the art will appreciate that embodiments of the inventionmay be interfaced to and controlled by a computer readable storagemedium having stored thereon a computer program. The computer readablestorage medium includes a plurality of components such as one or more ofelectronic components, hardware components, and/or computer softwarecomponents. These components may include one or more computer readablestorage media that generally stores instructions such as software,firmware and/or assembly language for performing one or more portions ofone or more implementations or embodiments of a sequence. These computerreadable storage media are generally non-transitory and/or tangible.Examples of such a computer readable storage medium include a recordabledata storage medium of a computer and/or storage device. The computerreadable storage media may employ, for example, one or more of amagnetic, electrical, optical, biological, and/or atomic data storagemedium. Further, such media may take the form of, for example, floppydisks, magnetic tapes, CD-ROMs, DVD-ROMs, hard disk drives, and/orelectronic memory. Other forms of non-transitory and/or tangiblecomputer readable storage media not list may be employed withembodiments of the invention.

A number of such components can be combined or divided in animplementation of a system. Further, such components may include a setand/or series of computer instructions written in or implemented withany of a number of programming languages, as will be appreciated bythose skilled in the art. In addition, other forms of computer readablemedia such as a carrier wave may be employed to embody a computer datasignal representing a sequence of instructions that when executed by oneor more computers causes the one or more computers to perform one ormore portions of one or more implementations or embodiments of asequence.

This written description uses examples to disclose the invention,including the best mode, and also to enable any person skilled in theart to practice the invention, including making and using any devices orsystems and performing any incorporated methods. The patentable scope ofthe invention is defined by the claims, and may include other examplesthat occur to those skilled in the art. Such other examples are intendedto be within the scope of the claims if they have structural elementsthat do not differ from the literal language of the claims, or if theyinclude equivalent structural elements with insubstantial differencesfrom the literal languages of the claims.

What is claimed is:
 1. An imaging system comprising: a shiftable imageplane generated from photons emanating from an object and passingthrough a lens; a detector plane positioned to receive the photons ofthe object that pass through the lens; and a computer programmed to:characterize the lens as a lens function; acquire two or more elementalimages of the object with the image plane at different axial positionswith respect to the detector plane; determine a focused distance of theobject from the lens based on the lens function and the two or moreelemental images acquired; and generate a depth map of the object basedon the determined focused distance.
 2. The system of claim 1 wherein thecomputer, in being programmed to characterize the lens, is furtherprogrammed to characterize the lens as a function of a lens aberrationprofile and a point spread function (PSF) that is a response to pointsources that are positioned at different locations with respect to thelens.
 3. The system of claim 2 wherein the computer is programmed tomodel the PSF as a Fourier transform of a pupil function of the imagingsystem that is represented in a form of multiple polynomials.
 4. Thesystem of claim 3 wherein the multiple polynomials are Zernikepolynomials up to third order aberrations.
 5. The system of claim 1wherein the object is a finger and the two or more elemental images ofthe object include at least two patches of a fingerprint of the finger.6. The system of claim 1 wherein the computer is programmed to determinethe object distance using a power spectrum ratio between the twoelemental images.
 7. The system of claim 6 wherein, when the object is athree-dimensional object, then the power spectrum ratio is determinedbetween associated patches in two elemental images of the two or moreelemental images.
 8. The system of claim 6 wherein the computer isprogrammed to determine the power spectrum ratio using a Fouriertransform.
 9. The system of claim 6 wherein the computer is programmedto: calculate a first value of a pupil function that corresponds withthe first elemental image; calculate a second value of the pupilfunction that corresponds with the second elemental image; determine aratio of the first value of the function and of the second value of thefunction; and minimize a difference between: the ratio of the firstvalue and the second value; and the power spectrum ratio; wherein thedifference is minimized by mathematically searching for a distance fromthe object to the lens that achieves the minimization.
 10. A method ofimaging an object comprising: characterizing a lens as a mathematicalfunction; acquiring elemental images of the object with an image planeof the object at differing axial positions with respect to a detector;determining a focused distance of the image plane to the object based onthe characterization of the lens and based on the elemental images; andgenerating a depth map of the object based on the determination.
 11. Themethod of claim 10 wherein mathematically characterizing the lenscomprises mathematically characterizing the lens as the mathematicalfunction that is based on an aberration profile of the lens and based ona response to point sources that are positioned at different locationswith respect to the lens.
 12. The method of claim 10 wherein the firstfocused distance is a distance to a first patch of the object, andwherein the method further comprises: determining a second focuseddistance of the object plane to the object based on the mathematicalcharacterization of the lens and based on the elemental images, whereinthe second focused distance is a distance to a second patch of theobject; and generating the depth map using the first patch of the objectand the second patch of the object.
 13. The method of claim 10 whereinmathematically characterizing the lens further comprises: modeling apoint spread function (PSF) as a Fourier transform of the imaging systempupil function; and representing the pupil function as one or morepolynomials.
 14. The method of claim 13 wherein representing the pupilfunction further comprises representing the pupil function as one ormore Zernike polynomials up to third order aberrations.
 15. The methodof claim 10 comprising: determining an elemental image ratio of theelemental images using a ratio of a power spectrum as determined for twoof the elemental images; and determining the first focal distance of theimage plane of the object includes using the elemental image ratio. 16.The method of claim 15 comprising determining the elemental image ratiousing a Fourier transform of two of the elemental images.
 17. The methodof claim 15 comprising: calculating a first value of the mathematicalfunction that corresponds with a first elemental image of the elementalimages; calculating a second value of the mathematical function thatcorresponds with a second elemental image of the elemental images;calculating a mathematical function ratio of the first value and thesecond value; and minimizing a difference between the elemental imageratio and the mathematical function ratio by mathematically varying adistance from the object to the lens when calculating the first value ofthe mathematical function and the second value of the mathematicalfunction.